During usability testing at Paycor, I watched participants search for controls, move between screens, and struggle to find the functionality they needed. Sometimes they were unsure whether their changes had saved.
When I saw confusion, I asked follow-up questions to understand what was happening. One participant explained: "Ah, I did not see the [scroll bar] earlier."
Then came the post-test questionnaire.
In the January study, 100% (10 of 10) responded positively to the statement: "Overall, I am satisfied with how easy it is to use this application."
That difference caught my attention. How could the ratings be so positive when I had just observed so much difficulty?
What the ratings told me-and what they left unanswered
The questionnaire asked participants to evaluate the application overall. My observations captured specific moments within individual tasks.
Those are related, but they are different perspectives. A participant could struggle to find a control and still have a favorable overall impression of the application.
The positive ratings did not erase the problems. The problems did not automatically make the ratings dishonest or meaningless.
I needed both kinds of evidence to understand the experience.
Other researchers have seen this too
MeasuringU addresses this directly in "Do Users Fail a Task and Still Rate It as Easy?". One possible explanation is that participants do not always realize they have failed. If they believe they reached the correct outcome, a positive rating may make sense from their perspective.
That research concerns task-level ratings, while my Paycor questionnaire measured perceptions of the application overall. Still, it illustrates why observed performance and reported ease can differ.
Other influences may also affect responses.
Nielsen Norman Group describes social desirability bias: people may give answers they believe others will view favorably. It also discusses acquiescence bias, the tendency to agree with statements-a relevant consideration when a questionnaire uses agreement-based questions.
Visual appearance can influence perceptions as well. NN/g's discussion of the aesthetic-usability effect includes examples of appealing designs softening participants' reactions to usability problems.
These are possible explanations. I cannot say they caused the positive Paycor ratings.
Why I asked follow-up questions
Watching someone struggle identifies a moment worth investigating. A follow-up question helps explain it.
The participant's comment about the scroll bar clarified an important distinction: the control was present, but they had not noticed it. Its presence alone did not make it discoverable.
That explanation gave me something more useful than an assumption about why the participant had stopped or searched elsewhere.
Follow-up questions can explore a rating, too. "What made you choose that rating?" gives participants room to explain what they considered. NN/g recommends asking respondents why they selected a rating to better understand their thought process.
The purpose is to understand the answer, without trying to persuade someone to change it.
What this changed about interpreting the results
If I had reported only the positive questionnaire results, the team would have missed important usability problems.
This experience also reinforced the value of recording usability sessions. A video clip can show the hesitation, searching, and repeated attempts that a positive rating does not capture. At other organizations, I used recordings to help teams see and understand the difficulties participants experienced.
The observed behavior showed where participants struggled. Their comments helped explain those moments. Task success and time on task added evidence about performance. The questionnaire captured their overall perceptions.
Together, those findings gave the team a fuller picture than any single measure could provide.
When participants struggle and then give a positive rating, I see a reason to look more closely. What were they evaluating? What did they notice? What did they believe they had accomplished?
A rating is useful. Understanding the experience behind it is what makes it actionable.
Email: theresaw@columbus.rr.com
LinkedIn: theresa-wilkinson